hi to all,
I have encountered some memory desallocation problem.
Is there somebody that knows if Doors is known for having desallocation efficiency problems?
I have to treat a lot of modules, so I open them in read only, treat the data and close the module.
Modules sizes are between 100Mo and 1Go. Do you have an idea about the size needed in RAM to open a 1 Go Modules?
Do yo know the time needed to fully restitute the memory needed in RAM to open a Module
I have the same questions on dynamic allocation, for instance on SkipList.
When delete a SkipList, do you know the time need to restitue the RAM taken by it ?
Maybe is there some commands I ignore to restitute RAM more efficiently?
Thanks for your help
xavier
xavierDxl - Thu Mar 04 05:58:43 EST 2010 |
|
Re: Desallocation memory response time SystemAdmin - Thu Mar 04 08:38:03 EST 2010
Yes, DOORS is a memory hog, especially all strings will be left in memory as there is no garbage collection. Buffers can be deleted from memory, but often the effect is not so large, as you have to handle the data as strings at some point.
What fo you mean by 100 Mo or 1 Go??
As for skip lists: if you insert string values into skip lists, the individual strings will be in memory even though you delete the skip list.
Do not know about timings for these operations, but generally, the more memory you consume, the slower everything works.
Attached is an old Telelogic technical tip paper on DXL memory consumption.
Attachments
attachment_14442058_Guidelines4MemoryMgmtinDXL.pdf
|
|
Re: Desallocation memory response time xavierDxl - Thu Mar 04 08:45:30 EST 2010 SystemAdmin - Thu Mar 04 08:38:03 EST 2010
Yes, DOORS is a memory hog, especially all strings will be left in memory as there is no garbage collection. Buffers can be deleted from memory, but often the effect is not so large, as you have to handle the data as strings at some point.
What fo you mean by 100 Mo or 1 Go??
As for skip lists: if you insert string values into skip lists, the individual strings will be in memory even though you delete the skip list.
Do not know about timings for these operations, but generally, the more memory you consume, the slower everything works.
Attached is an old Telelogic technical tip paper on DXL memory consumption.
thanks for our answer.
"What fo you mean by 100 Mo or 1 Go??" => I mean that the size of the modules I handle vary from 100Mo to 1Go max, depending the module (some of them are about 100Mo, some others 500Mo and some others 1Go)
Thank you very much for your support
regards,
xavier
|
|
Re: Desallocation memory response time llandale - Thu Mar 04 14:05:22 EST 2010
An individual module isn't going cause serious degredation.
One problem we found is that residually opened modules stay open, such as when you follow a link or access dxlAttr or Layouts that do the same thing. I've got a small suite of 'close everything up' that when 'too many' modules are currently open, it closes all modules that were not open when the script began. I arbitrarily set 'too many' to 40.
Another is to be sure to open modules Invisibly; this is especially true if you have remote custom 'addins' defined. When visible, critical also is to bypass the default view and load the standard view. Routinely do this: Module mod = read(NameModFull, false, true).
String parsing is a hog. Avoid concatenation inside a loop. Look at this code. See the completely rediculous explosion in response when concatenating strings, between 10,000 loops and 12,000 loops; from a palty 2 seconds to almost 2 minutes. Whereas the Buffer solution has linear response.
/* Start Mem Use = 900mb
Results:
NumLoops = 20,000
109 mSecs, Wasted space = 108890
xxxx far too long
NumLoops = 12,000
78 mSecs, Wasted space = 60890
111,922 mSecs, Wasted space = 349215495
NumLoops = 10,000
62 mSecs, Wasted space = 48890
2,188 mSecs, Wasted space = 239429495
*/
pragma runLim, 0
int i
int NumLoops = 10000
int Wasted = 0
string Result = ""
string Temp
Buffer bufResult = create()
bool DoBuff = confirm("Do buffer Test")
// int TimeStart = intOf(today())
int TimeStart = getTickCount_()
for (i=0; i<NumLoops; i++)
{ if (DoBuff)
{ Temp = i ""
bufResult += " "
bufResult += i ""
Wasted += length(Temp) +1
}
else
{ Result = Result " " i ""
Wasted += length(Result)
}
}
print (getTickCount_() - TimeStart) " mSecs, Wasted space = " Wasted "\n"
delete(bufResult)
I'm suspecting that DOORS reserves a big string table space when it starts, and slows down drastically when that space is exhausted and more is requested, incrementally, from the OS.
|
|
Re: Desallocation memory response time SystemAdmin - Mon Mar 12 20:40:31 EDT 2012 llandale - Thu Mar 04 14:05:22 EST 2010
An individual module isn't going cause serious degredation.
One problem we found is that residually opened modules stay open, such as when you follow a link or access dxlAttr or Layouts that do the same thing. I've got a small suite of 'close everything up' that when 'too many' modules are currently open, it closes all modules that were not open when the script began. I arbitrarily set 'too many' to 40.
Another is to be sure to open modules Invisibly; this is especially true if you have remote custom 'addins' defined. When visible, critical also is to bypass the default view and load the standard view. Routinely do this: Module mod = read(NameModFull, false, true).
String parsing is a hog. Avoid concatenation inside a loop. Look at this code. See the completely rediculous explosion in response when concatenating strings, between 10,000 loops and 12,000 loops; from a palty 2 seconds to almost 2 minutes. Whereas the Buffer solution has linear response.
/* Start Mem Use = 900mb
Results:
NumLoops = 20,000
109 mSecs, Wasted space = 108890
xxxx far too long
NumLoops = 12,000
78 mSecs, Wasted space = 60890
111,922 mSecs, Wasted space = 349215495
NumLoops = 10,000
62 mSecs, Wasted space = 48890
2,188 mSecs, Wasted space = 239429495
*/
pragma runLim, 0
int i
int NumLoops = 10000
int Wasted = 0
string Result = ""
string Temp
Buffer bufResult = create()
bool DoBuff = confirm("Do buffer Test")
// int TimeStart = intOf(today())
int TimeStart = getTickCount_()
for (i=0; i<NumLoops; i++)
{ if (DoBuff)
{ Temp = i ""
bufResult += " "
bufResult += i ""
Wasted += length(Temp) +1
}
else
{ Result = Result " " i ""
Wasted += length(Result)
}
}
print (getTickCount_() - TimeStart) " mSecs, Wasted space = " Wasted "\n"
delete(bufResult)
I'm suspecting that DOORS reserves a big string table space when it starts, and slows down drastically when that space is exhausted and more is requested, incrementally, from the OS.
Louie: Are you able to share the technique you use to get a handle on what modules are currently open so that they can be closed to release memory?
Everybody: I'm running into an out of memory situation with a "for item in current Project do" loop that opens every module and loads every view. There are hundreds of modules and thousands of links to lots of other modules. Even though I close each module after I process it, after about 130 modules I hit the 2GB mark and DOORS dies. I suspect that other modules are being opened (and remain open) due to layout DXL in views and are not being closed. I have been careful to avoid strings and use buffers instead. Thoughts?
Thanks!
Bob3
|
|
Re: Desallocation memory response time SystemAdmin - Tue Mar 13 09:08:00 EDT 2012 SystemAdmin - Mon Mar 12 20:40:31 EDT 2012
Louie: Are you able to share the technique you use to get a handle on what modules are currently open so that they can be closed to release memory?
Everybody: I'm running into an out of memory situation with a "for item in current Project do" loop that opens every module and loads every view. There are hundreds of modules and thousands of links to lots of other modules. Even though I close each module after I process it, after about 130 modules I hit the 2GB mark and DOORS dies. I suspect that other modules are being opened (and remain open) due to layout DXL in views and are not being closed. I have been careful to avoid strings and use buffers instead. Thoughts?
Thanks!
Bob3
for m in database do {
...
}
where:
m is a variable of type Module
Operation
Assigns the variable m to be each successive open module (for which the user has read access) in the database.
or
for m in project do {
...
}
where:
m is a variable of type Module
project is a variable of type Project
|
|
Re: Desallocation memory response time SystemAdmin - Tue Mar 13 13:30:14 EDT 2012 SystemAdmin - Tue Mar 13 09:08:00 EDT 2012
for m in database do {
...
}
where:
m is a variable of type Module
Operation
Assigns the variable m to be each successive open module (for which the user has read access) in the database.
or
for m in project do {
...
}
where:
m is a variable of type Module
project is a variable of type Project
SeanF, thanks for your response.
My concern with this approach is that if there are a lot of modules, this loop could increase my script's runtime. You see, I am already interating through all the modules in the project. With this approach, I would be iterating through all the modules in the project (closing any that are open) as many times as there are modules in the project.
Is there a more nuanced way of only closing those modules that are open (visibly and invisibly), rather than iterating through all modules?
|
|
Re: Desallocation memory response time SystemAdmin - Tue Mar 13 13:53:21 EDT 2012 SystemAdmin - Tue Mar 13 13:30:14 EDT 2012
SeanF, thanks for your response.
My concern with this approach is that if there are a lot of modules, this loop could increase my script's runtime. You see, I am already interating through all the modules in the project. With this approach, I would be iterating through all the modules in the project (closing any that are open) as many times as there are modules in the project.
Is there a more nuanced way of only closing those modules that are open (visibly and invisibly), rather than iterating through all modules?
You could use a call to evalTop_ in startup.dxl to create a top context skip list with addModule and removeModule functions.
You could then set a database wide post open module trigger to call add module and a database wide post close module trigger to call removeModule
You would then always have a skip list of all modules which you could query using funtions such as find rather than a loop but I think this would be unnecessarily complicated so I would suggest just using the database loop.
|
|
Re: Desallocation memory response time llandale - Tue Mar 13 16:15:50 EDT 2012 SystemAdmin - Mon Mar 12 20:40:31 EDT 2012
Louie: Are you able to share the technique you use to get a handle on what modules are currently open so that they can be closed to release memory?
Everybody: I'm running into an out of memory situation with a "for item in current Project do" loop that opens every module and loads every view. There are hundreds of modules and thousands of links to lots of other modules. Even though I close each module after I process it, after about 130 modules I hit the 2GB mark and DOORS dies. I suspect that other modules are being opened (and remain open) due to layout DXL in views and are not being closed. I have been careful to avoid strings and use buffers instead. Thoughts?
Thanks!
Bob3
They've scolded me for posting too many actual deployed functions.
[1] Perhaps it doesn't matter, but before you start get a list of all originally opened modules
-
Skip g_skpOriginal = create()
-
for mod in database do{put(g_skpOriginal, mod, mod)}
[2] "periodically" close the residually open modules
-
for mod in database do
-
{ if (find(g_skpOriginal, mod)) continue // don't close one that was already open
-
.. save(mod)
-
.. close(mod) // or maybe close(mod, false) don't save
-
}
The problem becomes how often is "periodically". If you do it every module you will end up doing lots of background opens and closes, since a linked module is probably linked from somewhere else. I think my algorithm arbitrarily counts the open modules, and if there are 40 or more then close the residually open ones. This check is made after each module your script is mindfully opening.
My original algorithm recursively found modules in each folder, and when I switched folders I'd close ther residually open ones. I figured modules in the same folder typically have the same target modules (parent requirements) and the same source linked modules (test procedures).
I made a feeble effort at trying to prioritize which ones to close by remembering which ones have been opened a lot. Thus, commonly used modules would remain open. Didn't seem worth it.
-Louie
|
|
Re: Desallocation memory response time SystemAdmin - Wed Mar 14 05:47:40 EDT 2012 SystemAdmin - Tue Mar 13 13:30:14 EDT 2012
SeanF, thanks for your response.
My concern with this approach is that if there are a lot of modules, this loop could increase my script's runtime. You see, I am already interating through all the modules in the project. With this approach, I would be iterating through all the modules in the project (closing any that are open) as many times as there are modules in the project.
Is there a more nuanced way of only closing those modules that are open (visibly and invisibly), rather than iterating through all modules?
Hi Rob,
The loop
for m in database do {
...
}
only iterates over the open modules, not all the modules, so I am not sure why you think it is not fit for the purpose you describe (identifying open modules so they can be closed). Sorry if I am not understanding something in your request.
|
|